Implements filter mechanism and improves filter at backend side for AAS Repository #516

mdanish98 · 2024-11-05T13:37:19Z

Description of Changes

The current implementation of CrudRepository does filtering InMemory only. However, it should be done on the backend side only based on specific implementation.

This PR adds support for filtering (Pagination/Metamodel) at the backend only. This PR adds some useful, common, and necessary classes as generic so that the Filtering can be applied easily to other repositories and services.

In this PR, the Filter and filtering at the backend feature is defined as a generic way to be used by other modules but the implementation has been done only for the AAS Repository. The reason is that the size of the PR would be way big if all the modules were considered. After this PR other modules implementation will follow.

Affected methods:
getAll*() -> Pagination

Related Issue

#437, #532

Signed-off-by: Mohammad Ghazanfar Ali Danish <[email protected]>

…ry-crud

Signed-off-by: Mohammad Ghazanfar Ali Danish <[email protected]>

mateusmolina-iese · 2024-12-10T08:17:26Z

....common/basyx.core/src/main/java/org/eclipse/digitaltwin/basyx/core/BaSyxCrudRepository.java

+ * 
+ * @author danish
+ */
+public interface BaSyxCrudRepository<T> extends CrudRepository<T, String> {


A few notes:

consider using the Spring interface QuerydsIPredicateExecutor.

I think this generic interface may not be necessary. I would consider adding a class to the core backend of the repositories/services/registries themselves instead.

consider renaming this interface (or interfaces, if you think (2) is valid) to backend; the keyword repository is used with different semantics in the context of this project. We decided to use backend for the backend management of the Spring side - which happens to be called repositories. Maybe we can come up with a better naming convention later 😅

mateusmolina-iese · 2024-12-10T08:29:39Z

...core/src/main/java/org/eclipse/digitaltwin/basyx/common/mongocore/MongoDBCrudRepository.java

+ * 
+ * @author danish
+ */
+public class MongoDBCrudRepository<T> extends SimpleMongoRepository<T, String> implements BaSyxCrudRepository<T>  {


Probably not necessary with QuerydslPredicateExecutor. See querydsl-mongodb.

mateusmolina-iese · 2024-12-10T08:38:21Z

...src/main/java/org/eclipse/digitaltwin/basyx/aasrepository/client/ConnectedAasRepository.java

@@ -77,7 +78,7 @@ public String getBaseUrl() {
 	}

 	@Override
-	public CursorResult<List<AssetAdministrationShell>> getAllAas(PaginationInfo pInfo) {
+	public CursorResult<List<AssetAdministrationShell>> getAllAas(PaginationInfo pInfo, Filter filter) {
 		String encodedCursor = pInfo.getCursor() == null ? null : Base64UrlEncoder.encode(pInfo.getCursor());
 		return repoApi.getAllAssetAdministrationShells(null, null, pInfo.getLimit(), encodedCursor);


It's currently not matching the capabilities of the http controller. The two nulls could be retrieved from the filter object

mateusmolina-iese · 2024-12-10T08:39:21Z

...test/java/org/eclipse/digitaltwin/basyx/aasrepository/client/TestConnectedAasRepository.java

+	@Test
+	@Ignore
+	public void retrieveAllAasWithSpecificAssetIdAndIdShortFiltering() throws Exception {
+		//TODO: Clients doesn't support filtering, remove this when client has the functionality


the internal api supports it. See remark in ConnectedAasRepository.

mateusmolina-iese · 2024-12-10T08:40:26Z

...org/eclipse/digitaltwin/basyx/aasrepository/client/TestAuthorizedConnectedAasRepository.java

@@ -75,7 +77,7 @@ public void removeAasFromRepo() throws FileNotFoundException, IOException {
 		TestAuthorizedConnectedAasService.configureSecurityContext(TestAuthorizedConnectedAasService.getTokenProvider());

 		AasRepository repo = appContext.getBean(AasRepository.class);
-		repo.getAllAas(PaginationInfo.NO_LIMIT).getResult().stream().map(s -> s.getId()).forEach(repo::deleteAas);
+		repo.getAllAas(PaginationInfo.NO_LIMIT, new AasFilter()).getResult().stream().map(s -> s.getId()).forEach(repo::deleteAas);


AasFilter.NO_FILTER or simply passing null?

mateusmolina-iese · 2024-12-10T09:26:23Z

.../eclipse/digitaltwin/basyx/aasrepository/backend/mongodb/TestAasMongoDBFilterResolution.java

+ * 
+ * @author danish
+ */
+public class TestAasMongoDBFilterResolution {


Consider reusing the TestInMemoryAasFilterResolution by extracting a testsuite. The test coverage is considerably different for component with the same function.

mateusmolina-iese · 2024-12-10T09:29:18Z

...repository-core/src/main/java/org/eclipse/digitaltwin/basyx/aasrepository/AasRepository.java

@@ -50,7 +51,7 @@ public interface AasRepository {
 	 * 
 	 * @return a list of all found Asset Administration Shells
 	 */
-	public CursorResult<List<AssetAdministrationShell>> getAllAas(PaginationInfo pInfo);
+	public CursorResult<List<AssetAdministrationShell>> getAllAas(PaginationInfo pInfo, Filter filter);


Consider overloading this method. There are lot of instances of an empty filter being passed and this measure would reduce quite a bit the amount changes of this PR. Also make sure to document the @param

mateusmolina-iese · 2024-12-10T09:35:50Z

...yx.core/src/main/java/org/eclipse/digitaltwin/basyx/core/pagination/PaginationUtilities.java

+	    return resolveCursor(pRequest, data, Function.identity());
+	}
+
+	public static <T> String resolveCursor(PaginationInfo pRequest, List<T> data, Function<T, String> idResolver) {


I would consider adding these two abstract methods to PagginationSupport, since it already serves as a util class. Also consider leaving this refactoring to another PR. It's not necessarily related to the scope of the current's PR and increases quite a bit the amount of changes (when considering, for example, the changes in aasdiscoveryservice)

mateusmolina-iese · 2024-12-10T09:36:21Z

...rc/main/java/org/eclipse/digitaltwin/basyx/aasdiscoveryservice/backend/CrudAasDiscovery.java

@@ -56,7 +56,7 @@
 */
 public class CrudAasDiscovery implements AasDiscoveryService {


The changes here are not necessarily related to the PR. See remark in common.pagination

mateusmolina-iese · 2024-12-10T09:36:53Z

...est/java/org/eclipse/digitaltwin/basyx/aasdiscoveryservice/backend/CrudAasDiscoveryTest.java

@@ -34,7 +34,7 @@ public class CrudAasDiscoveryTest {

    @Test
    public void getConfiguredAasDiscoveryName(){
-		AasDiscoveryService service = new SimpleAasDiscoveryFactory(null, CONFIGURED_AAS_DISCOVERY_NAME).create();
+		AasDiscoveryService service = new SimpleAasDiscoveryFactory(() -> null, CONFIGURED_AAS_DISCOVERY_NAME).create();


not related to pr

mateusmolina-iese · 2024-12-10T09:39:42Z

basyx.common/basyx.core/src/main/java/org/eclipse/digitaltwin/basyx/core/Filter.java

@@ -0,0 +1,9 @@
+package org.eclipse.digitaltwin.basyx.core;


License header

mdanish98 added 8 commits October 16, 2024 17:01

Moves InMemory CrudRepository to inmemory core module

6a51b5c

Signed-off-by: Mohammad Ghazanfar Ali Danish <[email protected]>

Refactors *InMemory Crud Repo to common

47c8c18

Signed-off-by: Mohammad Ghazanfar Ali Danish <[email protected]>

Merge remote-tracking branch 'upstream/main' into fix/refactor-inmemo…

9686a99

…ry-crud

Removes NO_LIMIT_PAGINATION_INFO

f8ed64b

Signed-off-by: Mohammad Ghazanfar Ali Danish <[email protected]>

Implements feature for filtering elements at backend only

1dd8b8f

Signed-off-by: Mohammad Ghazanfar Ali Danish <[email protected]>

Resolves conflicts

b9b14c5

Signed-off-by: Mohammad Ghazanfar Ali Danish <[email protected]>

Resets CI files

68217b9

Signed-off-by: Mohammad Ghazanfar Ali Danish <[email protected]>

Refactors codes and add license headers

c231307

Signed-off-by: Mohammad Ghazanfar Ali Danish <[email protected]>

mdanish98 changed the title ~~Implements filter mechanism and improves filter at backend side~~ Implements filter mechanism and improves filter at backend side for AAS Repository Nov 5, 2024

mdanish98 added 2 commits November 5, 2024 14:41

Fixes CI

06c9bbc

Signed-off-by: Mohammad Ghazanfar Ali Danish <[email protected]>

Adds missing import

c404266

Signed-off-by: Mohammad Ghazanfar Ali Danish <[email protected]>

mdanish98 marked this pull request as ready for review November 5, 2024 14:00

Adds Javadoc and license headers to PaginationUtilities

2c7776e

Signed-off-by: Mohammad Ghazanfar Ali Danish <[email protected]>

zhangzai123 mentioned this pull request Nov 21, 2024

[BUG] pagination does not improve response time #532

Open

mateusmolina-iese self-requested a review December 5, 2024 12:16

mateusmolina-iese suggested changes Dec 10, 2024

View reviewed changes

mateusmolina-iese reviewed Dec 10, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implements filter mechanism and improves filter at backend side for AAS Repository #516

Implements filter mechanism and improves filter at backend side for AAS Repository #516

mdanish98 commented Nov 5, 2024 •

edited by zhangzai123

Loading

mateusmolina-iese Dec 10, 2024

mateusmolina-iese Dec 10, 2024

mateusmolina-iese Dec 10, 2024

mateusmolina-iese Dec 10, 2024

mateusmolina-iese Dec 10, 2024

mateusmolina-iese Dec 10, 2024

mateusmolina-iese Dec 10, 2024 •

edited

Loading

mateusmolina-iese Dec 10, 2024

mateusmolina-iese Dec 10, 2024

mateusmolina-iese Dec 10, 2024

mateusmolina-iese Dec 10, 2024

		@@ -56,7 +56,7 @@
		*/
		public class CrudAasDiscovery implements AasDiscoveryService {

Implements filter mechanism and improves filter at backend side for AAS Repository #516

Are you sure you want to change the base?

Implements filter mechanism and improves filter at backend side for AAS Repository #516

Conversation

mdanish98 commented Nov 5, 2024 • edited by zhangzai123 Loading

Description of Changes

Related Issue

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mateusmolina-iese Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdanish98 commented Nov 5, 2024 •

edited by zhangzai123

Loading

mateusmolina-iese Dec 10, 2024 •

edited

Loading